2002  Command  and  Control 
Research  and  Technology  Symposium 


Topic:  C2  Decision  Making  &  Cognitive  Analysis 
Team  Collaboration  for  Command  and  Control: 


A  Critical  Thinking  Model 


*Jared  T.  Freeman,  Ph.D. 
Aptima,  Inc. 
Washington,  DC 


Daniel  Serfaty 
Aptima,  Inc. 
Woburn,  MA 


*Contact:  Jared  Freeman,  Ph.D. 
Aptima,  Inc. 

1030  15th  St.,  NW,  Suite  400 
Washington,  DC  20005 
202-842-1548  x316  (voice) 
202-842-2630  (fax) 
freeman  @  aptima.com 


Report  Documentation  Page 

Form  Approved 

0MB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1.  REPORT  DATE 

2002 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2002  to  00-00-2002 

4.  TITLE  AND  SUBTITLE 

Team  Collaboration  for  Command  and  Control:  A  Critical  Thinking 
Model 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Aptima  Inc, 1030  15th  Street  NW,  Suite  400, Washington, DC, 20005 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

2002  Command  and  Control  Research  and  Technology  Symposium  (CCRTS) 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

8 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98} 

Prescribed  by  ANSI  Std  Z39-18 


Team  Collaboration  for  Command  and  Control: 
A  Critical  Thinking  Model 


Jared  T.  Freeman,  Ph.D. 

Aptima,  Inc. 

1030  15^^  St.,  Suite  400 
Washington,  DC  20005 
freeman  @  aptima.  com 


Daniel  Serfaty 
Aptima,  Inc. 

12  Gill  St.,  Suite  1400 
Woburn,  MA  01801 
serfatv@  aptima.com 


Abstract 

We  define  an  initial  framework  for  understanding,  measuring,  and  managing  one  form  of 
collaboration:  team  critical  thinking.  The  framework  will  be  used  to  understand  how  team 
members  critique  and  refine  team  performance,  develop  measures  of  performance,  and 
eventually  to  create  training  and  decision  aids  that  support  this  form  of  collaboration.  The 
framework  leverages  recent  research  and  theory  concerning  individual  critical  thinking, 
teamwork,  and  information- age  warfare.  A  sample  of  measures  is  presented. 


Introduction 

Great  attention  has  been  paid  to  the 
advanced  weapons,  communications  networks, 
and  information  systems  used  in  modem  battles 
and  experiments.  To  make  effective  use  of  these 
resources,  however,  warfighters  require 
effective  collaboration  tools  and  practices.  In 
this  sense,  collaboration  is  a  crucial  element  of 
information  age  warfare.  Analyses  of  a  recent 
Future  Joint  Forces  (FJF)  experiment  suggest 
that  warfighters  recognize  the  importance  of 
collaboration  technologies,  and  clearly 
distinguish  good  technologies  from  bad 
(Freeman,  et  ah,  2002).  These  data  have  led  FJF 
leadership  to  argue  that  improving  collaboration 
tools  and  their  use  may  provide  a  better  return 
on  investment  for  DOD  than  more  common 
hardware  acquisition  programs  (Brooks, 
Institute  for  Defense  Analysis,  personal 
communication,  March  2002). 

However,  systems  in  military  experiments 
and  the  commercial  sector  illustrate  that 
collaboration  tools  and  practices  are  often 
awkwardly  implemented  and  their  use  is  poorly 
understood.  A  meta-analysis  of  laboratory 
studies  comparing  face-to-face  collaboration 


with  computer-mediated  collaboration  revealed 
that  computer  support  for  collaboration 
decreased  group  effectiveness,  increased  time  to 
task  completion,  and  decreased  team  member 
satisfaction  (Baltes,  Dickson,  Sherman,  Bauer, 
and  LaGanke,  2002).  A  meta-analysis  by  Dennis 
and  Wixom  (2001)  showed  that  decision  quality 
was  lower  in  virtual  teams  using  collaboration 
technologies  than  for  co-present  teams,  largely 
as  a  function  of  lower  process  facilitation  in  the 
virtual  condition.  There  is  clearly  room  for 
significant  improvement  in  the  development  and 
use  of  collaboration  techniques  and  the  tools 
that  support  them. 

To  detect  these  faults  and  support 
collaboration  better,  we  need  to  measure 
collaboration,  refine  the  technologies  and 
techniques,  then  measure  and  refine  again  and 
again.  To  develop  valid,  reliable,  and  useful 
measures,  we  must  understand  collaboration. 
However,  collaboration  has  proved  to  be  a  very 
subtle  phenomenon,  even  when  tools  are 
provided  to  constrain  and  shape  it.  Shirany, 
Tafti,  and  Affisco  (1999)  found  that  the  success 
of  collaboration  techniques  varies  with  task 
demands.  They  found,  for  example,  that 
experimental  teams  using  group  support  systems 


generated  more  ideas,  while  teams  using  basic 
email  performed  deeper  analyses  of  the 
problems  (as  indicated  by  a  greater  proportion 
of  inferences  drawn).  Maznevski  and  Chudoba 
(2000)  found  that  effective  teams  addressed 
such  issues  by  implementing  strategies  for 
selecting  face-to-face  or  remote  communication 
as  a  function  of  the  task  at  hand,  and  by 
developing  a  rhythm  of  communications  that 
interspersed  face-to-face  collaboration 
interspersed  with  remote  communication. 

The  inherent  complexity  of  collaboration 
has  produced  communities  of  research,  under 
flags  such  as  Situated  Cognition  and  Computer 
Supported  Cooperative  Work.  However,  there  is 
not  yet  a  coherent  body  of  theory  concerning 
collaboration.  There  is,  for  example,  no 
generally  accepted  mapping  of  team  attributes  to 
technology  and  tasks  (Christensen  and 
Fjermestad,  1997),  and  no  solution  to  the 
smaller  problem  of  fitting  collaboration 
technology  to  collaborative  tasks  (Zigurs  and 
Buckland,  1998).  Accordingly,  there  is  little  to 
guide  analysis  of  collaboration  and  its 
measurement  generally,  and  little  to  inform  its 
application  within  the  domain  command  and 
control  in  particular.  Yet,  collaboration  theory 
and  measures  are  crucial  if  we  are  to  achieve  the 
full  promise  of  information  age  warfare. 

We  are  initiating  a  program  of  research  to 
focus  on  one,  crucial  type  of  collaboration, 
which  we  call  team  collaboration  in  critical 
thinking  (TC  T).  In  this  program,  we  are 
drawing  on  three  relevant  research  threads: 

Recent  theory  and  fieldwork  concerning 
individual  and  collaborative  cognition  in 
command  and  control  and  other  information¬ 
intensive  organizations  has  provided  insights 
into  the  processes  by  which  team  members  may 
interpret  data  to  develop  information,  build 
understanding  that  informs  decisions,  and 
collaborate  to  ensure  that  information  and 
knowledge  are  shared  in  support  of 
synchronized  action  to  shape  events.  Alberts, 
Garstka,  Hayes,  and  Signori  (2001)  have 
developed  a  framework  that  clearly  defines 
these  (italicized)  constructs  as  primitives  of 
performance  in  information  age  warfare. 
Freeman,  et  al.  (2000)  conducted  initial 


validation  of  a  closely  related  model  of 
knowledge  management,  based  on  field 
observations  at  a  Marine  intelligence  operation 
and  re-analyzed  field  data  from  an  analysis  of 
Virtual  Information  Center  (VIC)  at  The  Pacific 
Command  (PACOM). 

Research  concerning  critical  thinking  has 
validated  a  theory  of  how  individual  warfighters 
make  decisions  under  uncertainty.  According  to 
the  recognition-metacognition  framework 
(Cohen,  Freeman,  and  Thompson,  1998;  Cohen 
and  Freeman,  1997;  Freeman,  Cohen,  and 
Thompson,  1998),  expert  warfighters  (1) 
monitor  for  opportunities  to  critique  their 
assessments  and  plans,  (2)  identify  sources  of 
uncertainty  (specifically  gaps,  untested 
assumptions,  and  conflicting  interpretations), 
and  (3)  reduce  or  shift  that  uncertainty  by 
gathering  information,  testing  assumptions, 
forming  contingency  plans,  etc.,  before  taking 
action. 

Research  conducted  under  ONR’s  A2C2 
program  and  other  Air  Force  Research 
Laboratories  human  engineering  projects  has 
produced  a  rich  body  of  measures  concerning 
the  processes  by  which  teams  coordinate  their 
activities  explicitly,  that  is,  through  explicit 
communication,  and  implicitly,  through  reliance 
on  shared  information,  shared  interpretations  of 
information  patterns,  and  standardized 
responses  to  those  patterns  (MacMillan,  Entin, 

&  Serfaty,  in  press;  Macmillan,  et  ah,  2001; 
Miller,  Price,  Entin,  &  Rubineau,  2001;  Moon, 
et  ah,  2000).  In  addition,  this  research  has 
refined  the  methods  of  model-based 
measurement  and  model-based  experimentation, 
in  which  experimental  hypotheses  are  developed 
from  models,  measures  are  built  to 
operationalize  key  variables,  hypotheses  are 
tested  in  models,  then  tested  empirically,  and 
the  results  are  used  to  refine  models  for  the  next 
round  of  hypothesis  generation,  measure 
construction,  and  testing. 

Our  approach  is  to  weave  together  these 
three  research  threads  -  collaborative  cognition, 
individual  critical  thinking,  and  model-based 
measurement  of  team  performance  -  to  develop 
a  model  and  validated  measures  that  help  us  to 
understand  team  critical  thinking,  and 


eventually  to  predict  and  manage  this  form  of 
collaboration. 

The  TC^T  Model 

We  leverage  the  work  on  collaborative 
cognition  (above)  by  extending  the  framework 
proposed  by  Alberts,  et  al.  (2001)  to  draft  a 
TC2T  model.  That  framework  specifies  that 
information  age  warfare  involves  events  in  three 
domains  (represented  in  the  diagram  by  three 
strata):  the  physical  domain  of  actions  and 
events;  the  information  domain  in  which  events 
are  observed  and  communicated;  and  the 
cognitive  domain  in  which  the  warfighter 
develops  understanding  in  support  of  decisions 
that  determine  further  actions.  The  flow  is 
illustrated  in  the  left  portion  of  Figure  1. 

Collaboration  between  warfighters  is 
represented  by  overlap  or  intersections  between 
two  of  these  diagrams.  Several  points  of 
intersection  or  linkage  are  possible  (but  not 
illustrated  here).  Shared  (or  linked) 
understanding  represents  common  situation 
understanding  and  expectations.  Coordinated 
decisions  constitute  plans.  Coordinated  actions 
denote  synchronization.  In  Figure  1,  we  have 
overlaid  a  node  and  links  to  represent 
collaborative  critical  thinking,  a  special  case  of 
collaboration.  Collaborative  critical  thinking 
resides  partly  in  the  information  domain  because 
it  consists  of  practices,  technologies,  team 
architectures,  and  other  enablers  that  support  the 
sharing  of  information.  It  resides  partly  in  the 
cognitive  domain  because  the  content  that  is 
shared  and  critiqued  includes  individual 
understanding  and  decisions  as  well  as 
observations  of  effects \ 


'  The  notion  that  cognition  exists  outside  the  mind  of 
the  individual  stretches  traditional  definitions  of  the 
term.  The  case  for  interpreting  cognition  in  this 
manner  has  been  well  argued  by  Hutchins  (1995a), 
who  demonstrated  that  cognitive  acts  are  afforded 
and  constrained  by  external  entities  and  objects. 

Thus,  shipboard  navigation  is  seen  as  a  cognitive 
process  that  results  from  coordination  between  a  team 
of  people  and  by  their  use  of  navigation  instmments. 
Hollan  et  al.,  (2000)  has  elaborated  this  view,  arguing 
that  cognition  unfolds  not  just  by  the  manipulation  of 
symbols  within  the  mind  of  an  individual,  but  through 
interpersonal  exchanges  and  manipulation  of  objects. 


To  define  the  processes  of  collaborative 
critical  thinking,  we  turn  to  the  recognition- 
metacognition  framework  of  critical  thinking 
(above)  proposed  and  validated  by  Cohen,  et  al. 
(1997,  1998)  in  studies  of  critical  thinking  by 
individual  warfighters.  Team  members  monitor 
the  state  of  the  mission  and  the  team  to 
discriminate  those  problems  that  are 
satisfactorily  addressed  with  rapid,  recognitional 
decision-making  (c.f.,  Klein,  1993),  from  those 
that  require  more  effortful  (and  time- 
consuming)  critiques.  Monitoring  triggers  a 
critique  of  understanding  when  (1)  uncertainty 
is  high,  (2)  stakes  warrant  high  confidence  or 
accuracy,  and  (3)  time  is  available  for  a  critique. 
That  critique  should  uncover  gaps  in 
knowledge,  conflicting  interpretations  of  the 
evidence  at  hand,  and/or  untested  assumptions. 

It  may  foster  decisions  and  subsequent  actions 
to  passively  wait  out  the  development  of  the 
situation,  or  actively  shape  the  battle  to  acquire 
information  or  advantage. 

A  collaborative  extension  of  this  model 
addresses  a  number  of  new  issues.  For  example, 
it: 

•  Acknowledges  and  specifies  the  role  of 
environmental  enables  to  collaboration, 
such  as  technologies,  team  processes,  and 
team  architecture.  These  enablers  make 
topics  of  TC  T  more  (or  less)  salient  and 
actionable; 

•  Addresses  the  ways  in  which  team 
members  raise  and  resolve  issues 
deserving  of  critical  thought  across  the 
team; 

•  Defines  how  team  members  coordinate 
their  actions  to  resolve  or  shift  uncertainty, 
by  filling  gaps  in  knowledge,  resolving 
conflicting  assessments,  and  testing  or 
replacing  assumptions  and  hypotheses; 

•  Define  the  ways  in  which  teams 
disseminate  the  results  of  their 
collaboration  and  monitor  their  impact. 

•  Defines  planning  processes  by  which 
members  identify  the  issues  that  should  be 
monitored  and  the  range  of  acceptable 
uncertainty  or  deviation  from  a  norm; 

Analytical  and  empirical  studies  under 
ONR’s  A2C2  research  program  and  related  Air 


Force  Research  Laboratories  projects  have 
produced  a  rich  understanding  of  the 
relationship  between  team  process,  structures, 
and  outcomes,  and  methods  of  measuring  all  of 
them.  These  measures  address  a  host  of 
processes  relevant  to  collaboration  including 
information  sharing,  information  transfer  via 
push  and  pull,  specific  measures  of  coordination 
communications,  negotiation  over  shared  assets, 
task  delays  due  to  resource  contention,  etc.  The 
program  has  also  produced  useful  measures  of 
the  effects  of  collaboration,  such  as  level  of 
synchronization  in  time  and  coordination  of 
asset  allocation  relative  to  the  threat  (e.g.,  the 
value  of  munitions  relative  to  the  value  of  the 
threat).  These  and  other  measures  from  the 
literature  form  the  foundation  of  a  basic 
measurement  system  that  addresses  four  issues: 

•  Collaboration  Technology  Measures  of 
Performance  (MOPs)  address  the  impact 
of  selected  technologies  on  the  occurrence 
and  quality  of  TC^T. 

•  TC  T  MOPs  measure  the  occurrence, 
latency,  and  quality  of  collaborative 
critical  thinking. 

•  MOPs  assess  the  impact  of  TC^T  on 
team  information  management,  shared 
understanding,  coordination,  and  decision- 
making  concerning  the  C  environment. 

•  Measures  of  Effectiveness  (MOEs)  tap  the 
impact  of  TC  T  on  battlefield  events,  team 
structure,  and  team  process. 

The  relationship  of  the  measures  to  the 
constructs  in  the  draft  model  is  illustrated  in 
Eigure  2.  Representative  measures  are 
documented  in  Table  1. 

Summary 

In  sum,  we  are  developing  a  framework  for 
understanding  team  collaboration  in  critical 
thinking.  This  work  draws  on  several  extant 
threads  of  research  and  is  intended  to  produce 
useful  measures  of  this  phenomenon  and,  in 
time,  tools  for  training  and  supporting  the 
application  of  this  skill  in  C  environments. 
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Figure  1:  A  preliminary  model  of  Team  Collaboration  for  Critical  Thinking. 


Figure  2:  The  relationship  of  four  types  of  measures  to  the  model. 


Table  1:  Representative  measures 


Type  of  measure 

Measure 

Defiuitiou 

Source 

Collaboration 
technology  MOPs 

Reach:  Team 
connectivity 

Interconnectivity  of  team  members  to  each  other 
(“Communities  of  interest”)  in  Phase  I  report 

Alberts,  et  al.  (2001) 

Reach:  Information 
connectivity 

Interconnectivity  of  team  members  to  information 
sources  (“Information  access”)  in  Phase  I  report 

Reach:  Translation 

Degree  to  which  tool  supports  translation  between 
representations,  domains,  and  languages 

Richness:  Structured 
problem  representation 

Degree  to  which  the  tool  requires  structured 
representations  of  the  problem  at  hand 

Freeman,  et  al.  (2000) 
(Phase  I  report) 

Richness:  Deconfliction 

Degree  to  which  the  tool  supports  coordination  of 
activities  via  a  shared  workspace 

TCT  MOPs 

Planning  for  TC'T 

Collaboration  communications  (e.g.,  paraphrasing 
others,  explicit  statement  of  monitoring  plans  & 
critieria)  during  planning  stage. 

MacMillan,  Entin,  & 
Serfaty  (in  press) 

Monitoring 

Number  of  critiques  initiated  concerning  high  priority 
issues 

Cohen,  Freeman,  and 
Thompson  (1998) 

Diagnosis 

Number  of  gaps,  conflicts,  and  untested  assumptions 
identified 

Action 

Instances  of  probing  own  resources  for  data,  testing 
enemy  or  environment  for  data,  intentionally  waiting 
out  problem 

MOPs 

Shared  situation 

awareness 

Degree  to  which  team  members  share  memory  for 
current  location  of  objects  in  the  tactical  picture 

Endsley,  1988 

Shared  predictions  of 
future  situation 

Degree  to  which  team  members  share  predictions  of 
the  location  of  objects  in  the  tactical  picture 

Shared  situation 

assessment 

Degree  to  which  team  members  share  assessments  of 
the  intent  of  entities  of  operational  interest 

Cohen,  Freeman,  and 
Thompson  (1998) 

Mutual  awareness  of 
goals 

Degree  to  which  team  members  express  shared  goals 

n.a. 

Mutual  awareness  of 
information  needs 

Degree  to  which  team  members  anticipate  the 
information  needs  of  teammates 

Freeman,  Entin,  et  al. 
(2002);  Entin  &  Entin 
(2000) 

Mutual  awareness  of  next 
action(s) 

Degree  to  which  team  members  anticipate  the  actions 
of  teammates 

Mutual  awareness  of 
workload 

Accuracy  with  which  team  members  estimate  the 
subjective  workload  of  teammates 

MOEs 

Synchronization: 

Allocation 

Proportion  of  tasks  executed  without  conflicts  in 
resource  allocation 

Diedrich,  et  al.,  (2002) 

Synchronization: 

Execution 

Proportion  of  tasks  requiring  coordination  that  are 
successfully  executed 

MacMillan,  Entin,  & 
Serfaty  (in  press) 

Synchronization: 

Precision 

Lag  in  readiness  for  execution  of  synchronized  events 
between  first  and  last  operator  ready  to  act. 

Diedrich,  et  al.,  (2002) 

Effects:  Enemy  losses 

Proportion  of  targets  or  threats  destroyed 

Traditional 

Effects:  Friendly  losses 

Proportion  of  friendly  forces  destroyed  or  lost 

Effects:  Delay  in  enemy 
ops 

Latency  in  enemy  maneuvers  or  actions  (e.g., 
bombing  targets)  due  to  friendly  actions 

Traditional 

Effects:  Team  process 

Average  delay  in  task  execution  relative  to  optimal 
task  execution  schedule 

MacMillan,  Paley,  et 
al.  (2001) 

